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ABSTRACT 

This report describes a technique for establishing 
relative priorities among policy issues that have been systematically 
identified in and extracted from written documents. The method 
involves assigning values to five factors that affect an issue's 
priority. These factors are: uncertainty, or the degree to which 
information is lacking; imminence, or the degree to which responses 
must be made quickly; the probability of change in the policies or 
practices affected by the issue; the scope of the change; and 
sensitivity to information, or the degree to which new information 
can alter decisions. Each factor is given values on a scale from 1 to 
3 (low to high degree of irat.?ct), and the values are simply 
multiplied by each other to provide a score. The report reviews the 
background to the development of the technique, discusses briefly the 
identification of policy issues, describes the reaso:;ing behind 
selection of the factors incorporated in the technique, considers 
modification of the technique by weighting critical factors, comments 
on assessing the values to be assigned to'the factors, provides two 
examples of application of the method, and notes strengths and 
weaknesses of the process. An appendix provides further information 
on the system for identifying and prioritizing policy issu2S as 
developed by the Statistical Analysis Group in Education. (POD) 
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Preface 



The National Center for Education Statistics created the Statis- 
tical Analysis Group in Education (SAGE) to fulfill its mandate to 
produce policy 'relevant analyses of education survey data. The pur- 
pose of Task 1 of SAGE is to develop a system for the empirical deter- 
mination of policy issues in elementary and secondary education. To 
accomplish this purpose , efforts on this task are directed toward 
developing methods for (1) the identification of important policy 
issues y (2) the establishment of priorities among the identified 
issues, and (3) the specification of data and analysis needs to 
address the high priority issues* The present paper focuses on the 
second concern — the establishment of priorities. 

We would like to express our appreciation to the following indi- 
viduals who carefully reviewed and critiqued earlier drafts of this 
paper and contributed substantially to its present form: Peggie 
Campeau, William Clemans, Don McLaughlin, and Bob Rossi* We are also 
indebted to Lee J* Cronbach who shared a working draft of his manuscript 
Designing Educational Evalua ions with us. The ideas discu. sed in his 
manuscript were very influential in setting a framework for this 
paper. However, the authors take full responsibility for the paper's 
content * 
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Background 



Purpose of Policy Issue Analysis (SAGE Task 1) 

In 1974, as a part of PL 93-380, Congress established th3 purpose 
of the National Center for Education Statistics (NCES) to be to: 

(1) collect, collate, and, from time to time, 
report full ani complete statistics on the condi- 
tion of education in the United States; (2) con- 
duct and publish reports on specialized analyses 
of the meaning and significance of such statis- 
tics; (3) assist state and local education agen- 
cies in improving and automating their statistical 
and data collection activities; and (4) review and 
report on educational activities in foreign coun- 
tries* 

The selection of statistics to collect and analyses to perform is 
limitless, and the value of the Center's contribution to American 
educatic • will be determined, in large part, by the choices that are 
made* To be most effective, the Center must be capable of reporting 
information that fills the needs of policymakers; and the needs of 
policymakers can be characterized in terms of information that is used 
to decide aaong policy alternatives* In order to carry out their 
mission, the Center staff must formulate their statistical plans and 
allocate their resources based or an awareness of key issues in Ameri- 
can education* Ouce these issues arc identified and their relative 
importance is made clear, the Center can formulate rational plans for 
data collection, analysis, and reporting in a truly effective manner. 

Recognizing the need for a systematic method to identify policy 
issues to supplement the informal suzrvey of policymakers' needs and 
the sporadic literature review on which planning has traditionally 
been based, NCES included efforts to develop such a method as part of 
the work of the Statistical Analysis Group in Education (SAGE) . This 
report is the second in a series describing the development of this 
method. The first report, entitled Framework for Policy Issue Identi- 
fication (McLaughlin, Russ-Eft, & Rubin, 1978), presented a framework 
for identifying and extracting issues from written documents. As 
described in that first report, the goal of the method is to produce a 
set of alternative options for a data collection and reporting plan. 
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Each option will include a rationale consisting of specific policy 
issues anJ indicators of the importance of those issues. This report 
describes a technique for establishing the relative priorities of the 
issues. Forthcoming reports will apply these methods, providing (1) a 
list of key issues J^r elementary and secondary education^ and (2) de- 
scriptions of analytical models and data bases that can be used to 
address the issue questions. 

Perspectives on the Policy Process 

From the literature in political science and organization theory, 
several different perspectives have emerged concerning the explanation 
and improvement of the policy process. To illustrate how divergent 
the perspectives can be, two extreme perspectives are described and 
contrasted. One perspective, which we shall call "mechanistic," views 
the policy proce&s as a series of rational, problem-solving steps, for 
which solution mechanisms can be developed. The mechanistic perspective 
provides both a description and a prescription of the attributes of 
the policy process. This perspective has been characterized by some 
authors as possessing certain distinctive features (Lindblom, 1959; 
Braybrooke & Lindblom, 1963; Allison, 1971; Mann, 1976). These features 
include (1) the clarification of values as distinguished from an 
analysis of alternative policies; (2) the distijiction of ends from 
means; (3) the development of ends that are comparable, though different 
(4) the lack of constraints in the ability to assign resources among 
competing ends; (5) the availability of complete iaformation and 
comprehensive analysis of the system and its components; (6j the 
allocation of resources according to marginal utility; and (7) the 
existence of decisionmakers who can decide and who can make allo- 
cations of resources. 



^Although the method is applicable to all of education, the Cen 
ter's greatest needs for issue identification are currently in 
elementary and secondary education. 



The contrasting perspective appearing in the literature is 
usually developed as a reaction to specific oversimplifications of the 
mechanistic approach. Proponents of this perspective claim that the 
political or bureaucratic constraints are such that the mechaaistic 
approach is invalid , infeasible, or botii. We will refer to the con- 
trasting perspective as "humanistic," because it emphasizes the human 
constraints faced by decisionmakers. In particular, the humanistic 
perspective recognizes that (1) values may not be distinguished from 
alternative policies; (2) ends or objectives may be adjusted to the 
available means, obscuring distinctions between ends and means; 
(3) ends or objectives may not be comparable; (4) consideration of 
alternatives and consequences is limited; and (5) information about 
the system and its components is limited. This perspective also 
emphasizes the following aspects of the policy process: (1) decisions 
usually require only small changes from the existing system; (2) problems 
are approached in a piecemeal fashion over time; and (3) many people 
and institutions are involved in the decisionmaking process. In fact, 
because of the serial and incremental nature of the policy process, 
with decisions predicated on previous decisions, a discrete "decision" 
by a decisionmaker or a group of decisionmakers may be difficult to 
identify. See Braybrooke and Lindblom (1963), Simon (1965), Allison 
(1971), and Mann (1976) for more detailed discussions of these points. 

Although these two perspectives differ, it is useful to recognize 
that they both contribute to understanding and improving policymaking. 
The proponents of the mechanistic approach continue to search for 
heuristic solutions to the problem of formulating policies that will 
have the most beneficial outcomes, and the proponents of the humanis- 
tic approach continue to search for problems in the application of the 
proposed heuristic solutions. As a result of this conflict, we can 
expect increasingly complex mechanistic models that take into account 
more of the human factors affecting policymaking. 

Whether one is an optimist or pessimist about the possibility of 
developing pl system for identifying and prioritizing issues in order 
to guide data collection and analysis and, as a result, to better 
serve polic3^akers, depends in part upon whether one tends more toward 
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the mechanistic or humanistic perspective. We believe that there is 
room for optimism, particularly if one is :>t^nsitive to the important 
aspects of the two perspectives. It is through awareness of the 
political and bureaucratic constraints faced by decisionmakers that 
data collection, analysis, and reporting plans can be developed that 
will better serve policymaking. Such awareness, however, can be 
improved through a disciplined and systematic analysis of factors that 
affect the utility of information to decisionmakers. 

To introduce the system for establishing issue priorities, it 
is necessary to briefly review the procedures by which issues will be 
identified. 



Identification of Policy Issues 

Several techniques for identifying important policy issues exist 
and have been used in similar efforts. These techniques include 
(1) analysis of the opinions of key actors in a policy arena, obtained 
from interviews or surveys (e.g., the critical incident technique 
[Flanagan, 1954] or the Delphi technique [Dalkey, 1969; Dalkey & 
Rourke, 1972; Sackman, 1975]), (2) analysis of information requests 
from staff of executive agencies or from congressional staff (see some 
discussion of this approach in the report by Wild, Fortna, and Knapp, 
1978), (3) analysis of written documents (see a discussion of one such 
approach in the paper by McLaughlin, Russ-Eft, and Rubin, 1978), and 
(4) identification of questions thought to be topical and important 
based upon personal contact with policymakers and conversation about 
issues. 

Each of these techniques attempts to probe the infoimation net- 
work of the policy arena. In developing a systematic approach to the 
identification of issues, a mixture of these techniques was used. 
Policy issues and important sources of information (both persons and 
documents) were identified through interviews with key educational 
policymakers. (See the report entitled Results of Interviews w ith 
Educational Policymakers: A Step in the Development of an Issue 
Identification System for Educational Policies , Statistical Analysis 
Group in Education, 1979.) McLaughlin, Russ-Eft, and Rubin (1978) 
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provided a general framework for identifying issues from written 
documents. Detailed guidelines were then developed for identifying 
and extracting issues. In particular, issues were identified from 
writtt^n documents through the use of certain "flags" or issue indicators. 
Flags included (1) the appearance of words indicating controversy or 
potential for controversy (e.g., problem, issue, debate), (2) reports 
of changes in the educational system, (3) proposals for changes in the 
educational system, and (4) the appearance of causal statements, 
beliefs, and findings implying a needed change. (A description of 
these procedures, as they are currently being used, appears in 
Appendix A.) Additional interviews will be conducted with key policy- 
makers, after completing the issue identification procedures using 
literature oo elementary and secondary education. These interviews 
will serve to validate the findings and indicate deficiencies in the 
method. 

Establishing Priorities Among Issues 

Outline of the Proposed Approach 

The principal objective in establishing priorities among issues 
is to identify information that is urgently needed. Identification of 
the audience or clients for the information and definition of what 
constitutes urgently needed, or at least useful, information is a 
necessary first step in the proposed approach. 

The primary clients for the work of NCES are these individuals at 
the federal level who are involved in developing and deciding educational 
policies and practices. Included in this group are individuals in the 
administrative branch who help design presidential proposals submitted 
to Congress and who determine the operating procedures for programs 
mandated by Congress. Also included are legislators and their staffs 
who assess the strengths and weaknesses of mandated programs. Although 
the audience for the work of NCES is broad, the needs of the primary 
clients are seen as most critical. Furthermore, it is likely that 
satisfying the needs of this primary audience will, to a great extent, 
serve the needs of the larger audience (e.g., state and local policy- 
makers and practitioners) . 



Informatioa is useful to primary clients v.o the extent that it 
can help them in developing and deciding educational policies and 
practices. Thus, the proposed approach aims to estimate the extent to 
which investigation of a given issue is likely to yield useful infor- 
mation. More specifically, the approach attempts to assign priorities 
to issues on the basis of their projected usefulness fcr decisionmaking. 
This projection can be considered as an analogy to what is intuitively 
done by managers in deciding among alternative data collection efforts. 
The proposed approach will identify the factors that would be considered 
by a good manager — one who is sensitive to the political process, as 
well as knowledgeable about available social-science tools. In ad(?ition, 
it will guide the user in determining the value of each factor for 
each issue. 

Our approach is based on discussions with members of the primary 
client group and with close observers of this group, and on our reading 
of the literature concerning the educational policy process. The 
factors that are identified and the ii.athod for combining these factors 
are the result of considerable efforts to understand at what times and 
for what purposes federal policymakers value information. We recognize 
that the suggested set of rules for assessing and combining factors is 
one of many possible alternatives; nevertheless, we believe that it 
provides a reasonable approximation of the processes undertaken by 
NCES's primary clients. 

Factors in Establishing Priorities 

From the point of view of an educational policymaker, the relative 
usefulness of information addressing a particular issue is a function 
of the level of uncertainty about the issue (i.e., how uncertain the 
policymaker is about the answer to the question posed as the issue) 
and the perceived importance of knowing the answer to the question. 
These two factors make independent contributions to the level of 
I^riority of an issue. 

Since policies are attempts to meet social needs, they should be 
based upon knowledge of the causes of these needs and confidence in 
proposed approaches to meeting these needs. Thus, one may expect that 
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the perceived need for information in an issue area is inversely 
related to a decisionmaker's confidence that the causes of a problem 
are understood and the practices are appropriate for solving the 
problem* Lack of confidence, which we call uncertainty ,* contributes 
greatly to a policymaker's appetite for information. All other factors 
held constant, the expected value of information contributing to an 
answer for an issue question increases directly with the level of 
uncertainty about the answer. One can think of a policymaker considering 
a set of alternative approaches to delivering a social service (e.g., 
maiastreaming handicapped students versus segregating them in a special 
school). The policymaker may be certain that one approach will be 
most effective; on the other hand, be or she may be uncertain as to 
the superiority of any one approach. In the latter case, information 
is needed if rational action is to be taken. 

Priority will also vary as a function of the perceived importance 
of an issue. Four factors contribute to this assessment of importance: 
(1) the imminenc e of the events in that issue area impinging upon the 
decisionmaker; (2) the probability of change in policy or practice as 
a result of those events; (3) the scope of change ; and (4) the sensitivity 
of the policy process to new information (or the probability that the 
direction of change can be influenced by information). All of these 
factors are included in what others have referred to as the timeliness 
and relevance of information (Caplan, 1977; Gilmartin & McLaughlin, 
1977; Horst, Nay, Scanlon, Wholey, 1974). 

Imminence refers to the time pressure surrounding the need for 
information. Many of the events that contribute to the development of 
educational policies are part oi a relatively predictable agenda. The 
dates of reauthorization hearings or staff reviews of federal programs 



*The term uncertainty has a similar use in information theory and Bayesian 
statistics. To say that a person is "certain" is to say that he or she 
attaches a subjective probability of 1.0 to an outcome and a probability 
of 0 to all other possible outcomes. As uncertainty increases, the prior 
probabilities attached to the alternative outcomes approach equality. 
Thus, the greatest uncertainty occurs when all possible outcomes are seen 
as being equally probable. 
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as the basis for new legislation are known well in advance. The 
schedule for legislative response to bills introduced by the admini- 
stration is less certain y however, and other important ev' nts ma> 
occur ^^ith little warning (e.g., the passage of the Jarvi i-Gann 
referendum, Proposition 13, in California). Nevertheles: , the 
pressure oJ time in reacting to events in an issue area i; ^ ely uO 
increase the importance of an issue and of information ab \ t thai 
issue. It is essential, therefore, to estimate future deav'ines 

Upcoming events may call the attention of decisionmaker t'> a 
particular issue area; yet interest in new information may b ^v^la- 
tively low because of a low probability of a change in pol. 
practice* Hearings nay be scheduled , yet it may be know 
existing policy will be continued. The expected attendance at such 
hearings will be low. In contrast, change in some other policy area 
may be very likely. Important events may have occurred that will 
require changes in existing practices. Dra itic budget cuts may be 
expected y or new interest groups may have formed that have disturbed 
the old alliances from which pi^'svious policies emerged. The greater 
the likelihood of change in p^aicies oc practices, the greater will be 
the importance of issues related to those policies or practices, and 
the greater will be the need for relevant information. 

The scope of the e xpected change determines the likely importance 
of a change and, therefore, substantially affects the value of relevant 
information. Although the scope of change may have economic, political, 
and social ramifications, it can be considered in ononetary terms. An 
expansion of an existing program or the development of a new one 
involves the costs of allocating new personnel, facilities, and materials 
to achieve the goals of the program. Conversely, the termination of a 
program may result in monetary savings to taxpayers, although the 
dislocation of those no 3onger employed in delivering services and the 
loss experienced by those no longer receiving services must be considered 
in calculating costs. The expected scope of change may also be conceived 
in terms of departures from established rationales for earlier policies 
and practices. While change in rationales may have small immediate 
monetary effects, such a change may indicate a future direction that 
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will substantially transform programs and practices. For example, the 
emergence of an emphasis on competency-based exams may indicate a 
reallocation of resources from programs that are supportive of high- 
achieving students to programs that remediate learning problems of 
low*achieving students. Other factors being equal, the greater the 
scope of change in an issue area, the greater will be the importance 
of issues in that area and the value of information addressing those 
issues. 

Finally, the likelihood that the direction of change can be 
influenced by information, or the sensitivity to information, is an 
importanl factor in establishing issue priorities. Sometimes a change 
in policy or practice is very likely. At the same time, it may be 
unlikely that new information will play a role in determining the 
nature of the change. For example, a new political coalition may 
shift majority opi*^ion in Congress from favoring an existing program 
to favoring one particular modification of it. The change is likely, 
the scope of change is relatively broad, and the time pressure for 
making the change is great; however, the impact of new information 
will be attenuated. Where political forces seem already to have 
shaped the future, policymakers' needs for information will be reduced, 
la similar circumstances, however, there may be a new block of undecided 
"voters." New problems in an issue area may have emerged, and old 
practices may hold no promise of solution. In such cases, the impact 
of new information may be amplified. A cogent report may "frame" or 
"set" a new problem-solving approach (Rein & White. 1977). Because 
the cogency of a report depends in part on its empirical validity, a 
coalition between p olicy and information experts can be particularly 
potent at this stage. Other factors being equal, as the predictability 
of change in policy or practice increases, the likelihood that infor- 
mation can influence the direction of change decreases. 

Method of Combining the Factors 

To compare the priori?:y of different issues, it is necessary to 
have a method for combining the factors into a single scale. In 
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principle, the scLieme for combination should approximate the subjec- 
tive combining of factors performed by the primary clients. 

We assume that the system of issue identification will provide 
lists of information needs that far exceed the resource capacity of 
the Center. Planners must ther select among various opportunities for 
contribution. Frequently, the choices will be among issues that are 
considered important on one factor but unimportant on others. Thus, a 
method for calculating the relative importance of factors or their 
tradeoffs is needed. 

Unless the tradeoffs can be calculated fairly precisely and 
uniquely (e.g., on a valid "ratio" scale), any attempt to mechanize 
this process must be viewed as only providing tentative guidelines for 
planning. If the method merely provides a ranking of issue j, for 
example, that will not be sufficient information to decide on resources 
to allocate to the highest priority issue, the next highest, and so 
on. As we shall discuss in a forthcoming report, for each high priority 
issue it will be necessary to provide the Center with alternative 
activities to address the issues. Also, the system must provide a 
brief description of what is to be gained, at what cost, for each 
activity. 

Arguments can be made that any one of the factors (uncertainty, 
imminence, probability of change, scope of change, and sensitivity) is 
the most important factor. Furthermore, the relative importance of 
the factors may vary dramatically for different issues at different 
times. To provide an estimation of the relative priority of issues 
and tc illustrate the proposed method » a numerical assignment function 
has been developed to obtain a single priority rating for each issue. 

Table 1 provides values for each of the factors. Refinement of 
the values is a task that will be accomplished together with the 
Center staff. As indicated, the numerical assignment iunction that is 
presented here is only a first approximation. 

Priority Ra.. Ing = Uncertainty x Imminence x Probability 
of Change x Scope of Change x Sensi- 
tivity 

PR = (U)(I)(pC)(sC)(S) 
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Table 1 
Factors and Their Values 



Fsccors 



Oefinicions of Values 



Values 



Uncertalncy (0) 



Ths nmmd for informscion is grssc. Confidancs in any one approach is 3 
lov* Expcrca acknowledge auch uncercaincy. 

Sotw infonacion is svailsblc and chare ia aoae consenaus. 2 

Much perauasive inforaacion ia available. Additional daca will contri- 1 
bure only alighcly co che preciaion of knovledge. 



Indnence (I) Nev legialaclon or regulationa will be forchcoming chia year. Clienta 
are facing upcoming deadlinea. A **hoc" copic haa eaerged and alchcagh 
ic do%an'c yec appear on dienc agendaa, it moat certainly will. 

I3aua wlXl probably receive attention within the next tvo yeara. 

laaue doea not now appear on client agendaa, and th^^.re ia no indication 
it will in the near future. 



Probability of Probability of change in policy or practice (to which iaaue ia relevant) 
Change (pC) ia great. Budget aurplusea or deficit a have occurred; important find* 

ingfi have changed rationalea for exiatit'^ policlaa. New iM>litical 

preaaurea will cause change. 

Probability of change ia moderate. Indicationa are that exiating poli- 
ci«a or practices will be aerioualy inveatigated, and.changea will be 
conaidered. 

Probability of change ia small. No reaaon to expect d.'^fferent policiea 
o'j practices than are already in place. 

Scope of Chang would involve major aoclal program or rationalea for federal ex- 

change (sC) penditure that will effect major programs in the future. Changes would 

be substsntlal in impsct. 

Change would involve moderate-sized programs or rationales that might 
influence a small set of other programs. Impsct of change would be 
moderste. 

Changes would involve social programs of relatively minor scope, or 
larger programs with changes being in the nature of marginal refinements. 



Sex;aitivity 
to information 



New probleaa In iasue ares have srisen, snd the consensus is that new 
solutions must be found. There is s bloc of influential, but aa yet 
"undecided,** persons receptive to new information. The need for a new 
"framework" for approaching policy decisions is recognised. 

Given persuasive new information, established views may be swayed. 
Policymakers will continue paat policiea if no new information ia avail- 
able, but they will conaider new information should it emerge. 

Possible actiona with respect to policies snd prsctices to which isuues 
are relevant sre already predictsble with high confidence. Politicsl 
forces will dominate policymaking. 



*Note on imminence; Time pressure may be very great in some issue areaa, ao great that moat 
data collection methoda would be too time-consuming to yield information when needed. Still, in 
thia scheme, the iaauea would receive high weight a on imainence. The iaaue would then be ranked 
high In priority. We feel that auch ranking is sppropriate. Managera making allocation dec^.- 
siona will recognize that only "quick and dirty** methoda will yield timely resulta. They may 
chooae or reject auch methoda baaed on an aaaeaament of the expected value of the information; 
that ia, the degree to ^'hich auch information reducea uncertainty. After the event a that create 
the time preaaure have paaaad, the priority of the iaaue will be lowered. 
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BEST COPY AVAILABLE 



It should be noted that different methods for combining factors 
can be included in this simple multiplicative model. First, differential 
weights (weighting ixmninence more than sensitivity, for example), can 
be simulated by widening the intervals between numerical values on 
some factors relative to others. For example, changing "imminence" 
values from 1, 2, and 3 to 1, 4, and 9, by squaring them, would effectively 
double the weight of this factor relative to others.^ Second, for 
choices in which one alternative "dominates," or has a higher rating 
on all factors, the method of combining factors will be immaterial. 
Note that, in this multiplicative model, each of the factors can have 
a "zero" value (absolutely no imminence, no uncertainty, etc.). For 
issues with a zero value on one factor, the values on the other factors 
are irrelevant. 

Much remains to be done to refine the quantitative aspects of 
this method, including analysis of actual decisions. We feel that 
this can be accomplished most efficiently through a joint effort of 
individuals at the Center and in SAGE. 

Assignment of Factor Values 

Knowledge of the political context of an issue is needed for 
assigning values to the five factors. Such knowledge can be gained 
directly, through participation in the education policy process, or 
indirectly, through careful reading and close observation of that 
process. Since we are concerned here with developing an indirect 
approach, we must identify key indicators of factor values that can be 
monitored. (Those who participate directly in the policy process may 
monitor these indicators also.) 

From reading the literature in au area, one gains a perspective 
on the current controversies, the alternative positions, and the 
information available to support these positions. In terms of our 
proposed approach, unless some concern is expressed or some change is 



*If ^'^i^2^3^U' V=log + log + log + log C^; and squaring 

Cj effectively doubles the weight applied to log . 
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reported or recommended, no issue will be identified from the litera- 
ture. Because issues are seldom raised about social programs that 
have widespread confidence, our procedure for identifying issues will 
usually lead to the assignment of a high uncertainty rating. An 
important consideration in assigning a value to the factor of uncer- 
tainty is the degree to which alternative proposals exist for addressing 
an issue. A second consideration is the degree to which information 
exists to support proposals. If no alternative proposals (or positions) 
exist but concerns have been raised, uncertainty is likely to be high. 
If alternatives exist but none have much credibility because little 
supporting information exists, then uncertainty is again high. However, 
if alternatives exist and one of them is persuasively supported by 
information, the level of uncertainty must be lower. In such cases, 
depending oo the number of alternative positions and the amount and 
quality of their supporting information, the value assigned to the 
uncertainty factor would vary from intermediate to low. As the number 
of positions that appear to be persuasively supported increases, 
uncertainty again increases. 

The factor of imminence depends largely on the legislative calendar. 
Expiring legislation, leading to reauthorization and budget approval 
hearings, provides one important set of indicators. Other indicators 
may be contained within the legislation itself; in some cases, admini- 
strative actions with specific deadlines are mandated. Presidential 
statements and administrative regulations or practices that set certain 
daises for acrion provide another set of indicators. Timeframes and 
deadlines for action may also be set by court cases and public elections. 
The time constraints can be determined through careful reading of 
daily, print media, such as Education Daily , the Washington Post , and 
the New York Times . Media sources may not provide a clear statement 
of an issue, but they do serve a key function, especially Education Daily , 
in disseminatrng information about the timeframes within which important 
education-policy events will occur. 

Probability of change can also be determined from an analysis of 
the printed media. In assigning a value to the probability of change, 
it is important to go beyond an indentif ication of alternative proposals 
or positions and to determine the likelihood that any of the proposed 
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God, we will obey God" (Education Daily > December 11, 1978). In such 
cases, sensitivity to information may be low. 

Application of the Method 

To begin to provide the basis for evaluating the abstract concepts 
discussed above, we present two applications of the proposed method. 
The two issues that are considered here were identified and described 
in a report by Russ-Eft, Rubin, and Holmen (1978). These issue areas 
are: 

1. Explanations and solutions for low recruitment and retention 
rates in adult basic education (ABE) 

Which of several strategies for increasing the recruitment and 
retention rates are most effective? 

2. Counseling 

In what roles are counselors most cost-effective as members of 
the ABE staff? 

First, for the issue of low ABE recruitment and retention rates, 
we will examine each of the factors that determine its priority. 
Several explanations and proposed "solutions" for low recruitment and 
retention rates have been suggested, but no definitive investigations 
have examined all the alternatives. The alternatives seem equally 
plausible, thereby decreasing confidence that the best solution to the 
problem is known. Therefore, the uncertainty is high (U = 3). This 
issue, furthermore, formed the basis for some of the recent discussions 
on the amendments to the Adult Education Act. Although oome decisions 
were made and certain changes were incorporated into the Act, the 
issues related to recruitment and retention remain timely. Administrative 
officals in the U.S. Office of Education are currently drafting regulations 
for this new legislation. Information on these issues, especially if 
provided before May 1979, may affect the formulation of these regulations. 
Thus, imminence is high (1=3). Clearly, change in existing practices 
will occur as a result of the new legislation, tor example, emphasis 
is being placed on providing support services, such as transportation 
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and child care. The probability of change, therefore, is high (pC = 3). 
Change from previous practice may be fairly substactial, although the 
degree of change may vary from state to state. Programs in certain 
states, such as California, have already instituted special services 
to encourage increased recruitment and retention, while programs in 
other states have few such services. Therefore, the scope of the 
change is considered moderate (sC = 2) . xhe uew legislation has 
already determined the directions for the changes. The exact form 
that they will take, to be determined largely through the drafting of 
administrative regulations, has yet to be decided. Thus, the sensi- 
tivity to information is moderate (S = 2), The level of priority for 
this issue would be 

PR = (U)tI)(pC)(sC)(S) 
PR = (3) (3) (3) (2) (2) 
PR = 108 

In contrast, consider the issue on the cost-effectiveness of 
counselors. Questions have been raised concerning the role of counselors 
in ABE programs (e.g., Hezirow, Darkenwald, & Knox, 1975; O'Keefe, 
1976). Again, the level of uncertainty is high (U = 3). Given that 
these issues were not raised in the recent amendments, they are not of 
immediate concern. Since neither legislative nor regulatory changes 
are scheduled in the next year and major education amendments will not 
appear until 1984, the imminence is low (1=1). It is unlikely that 
major changes in programs would occur without some legislative mandate. 
Thus, the probability of change given any set of results on the issue 
is not very high (pC = 1). The scope of a change^ if it occurred, 
would be of moderate degree (sC = 2). Finally, since no major interest 
groups are committed on this issue, the sensitivity to information is 
moderate (S = 2). Therefore, the rating on this issue is as follows: 

R = (U)(I)(pC)(sC)(S) 

R = (3)(1)(1)(2)(2) 

R = 12 



ERLC 



Strengths and Weaknesses of the Approach 

We believe that use of the proposed method can be of substantial 
value to the Center in prioritizing issues for input to the process of 
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allocating resources. Because the system will specify uniform procedures, 
its application will ensure that all candidate issue*based data collection 
and reporting proposals will receive uniform attention during planning. 
This provides a safeguard against the tendency to overlook important 
elements in an unstructured assessment of the priority of issues. 

The method also provides a foundation for planning that can be 
used to supplement more subjective judgments of importance. Intuition 
and judgment must be a part of any prioritizing scheme, but total 
reliance on such subjective methods may sometimes cause planning to 
"miss the mark." When the proposed system is applied, there will no 
doubt be instances where the ratings that are determined will seem 
counterintuitive. These instances should alert decisionmakers \o 
probe for the causes of the discrepancies. The resulting conclusions 
may indicate needed refinement of the method or a reconsideration of 
previous judgments. 

The method is not without weaknesses. Its objective is to project 
the value of alternative sets of data and analyses for the primary 
audience for NECS efforts. Any such method, including ours, must make 
certain assumptions, and these assumptions are open to discussion and 
debate. In particular, the assumptions may have oversimplified 
(1) the definition of the primary NCES audience, (2) the identification 
of the factors that make information important to this audience, and 
(3) the determination and combination of the values for these factors 
for a particular issue. Another problem is that the determination of 
factor values may be difficult at times. For example, to rate proba- 
bility of change or sensitivity to information requires substantial 
knowledge about persons and groups who are politically active in the 
debate of an issue. When issues are emerging or changing rapidly, 
this knowledge may have to be acquired first-hand. 

A realistic assessment of the merit of the proposed method should 
not overlook these weaknesses. Still, we are optimistic that the 
strengths of the method outweigh the weaknesses. This method will, 
then, contribute to improvements in planning for organizations such as 
the Center that collect, analyze, and disseminate information. 



17 



References 

Allison ) 6. T. Essence of decision: Explaining the Cuban missile crisis . 
Boston: LittJf::, Brown, 1971. 

Braybrooke, D., & Lindblom, C. E. A strategy of decision: Policy evalua - 
tion as a social process . New York: Free Press, 1963. 

Caplan, N. Social research and national policy: What gets used, by whom, 
for what purposes, and with what effects. Evaluation Studies Review 
Annual , 1977, 2, 351-359. 

Dalkey, N. C. The Delphi method: An experimental study of group opinion 
(RM-5888-PR) . Santa Monica, Calif.: The Rand Corporation, 1969. 

Dalkey, N. C. , & Rourke, D. L. Experimental assessment of Delphi procedures 
with group value judgments. Studies in the quality of life . Lexington, 
Mass.: Lexington Books, 1972. 

Eidenberg, E. , & Morey, R. D. An act of Congress: The legislative process 
a nd the making of education policy . New York: Norton, 1969. 

Flanagan, J. C. The critical incident technique. Psychological Bulletin , 
1954, 51, 327-358. 

Gilmartin, K. J. , & McLaughlin, D. Classification of relevant evaluation 
methods (AIR-63500-6/77-FR) . Palo Alto, Calif.: American Institutes 
for Research, 1977. 

Horst, P., Nay, J. N., Scanlong, J. W. , & Wholey, J. S. Program management 
and the federal evaluator. In Public Administration Ke^i ew, 1974, 
34(6), 300-308. 

Lindblom, C. E. The science of "muddling through". Public Administration 
Review, 1959, 19, 79-88. 

Mann, D. Policy decision-making in education: An introduction to calcula - 
tion and control . New York: Teachers College Press, 1976. 

Mezirow, J. D., Darkenwald, G. D. , & Knox, A. B. Last gamble on education : 
Dynamics of adult basic education . Washington, D.C.: Adult Education 
Association, 1975. 

McLaughlin, D. H. , Russ-Eft, D., & Rubin, D. P. Framework for policy issue 
identification (Interim Report). Palo Alto, Calif.: Statistical 
Analysis Group in Education, American Institutes for Research, 1978. 

O'Keefe, M. The adult, education, and public policy . Paper presented at 
the Program in Education for a Changing Society of the Aspen Institute 
for Humanistic Studies, Aspen, Colorado, December 1976. 

Rein, M., & White, S. H. Policy research: Belief and doubt. Policy 
Analysis . 1977, 3, 239-271. 



19 



Russ-Eft, D., Rubin, D. P., & Holmen, R. Identification of issues in adult 
education (Draft Tech. Rep. No. 1). Palo Alto, Calif.: Statistical 
Anslysis Group in Education, American Institutes for Research, 1978. 



Sackman, H. Delphi critique: Expert opinion, forecasting, and group process . 
Lexington, Mass.: Lexington Books , 1975- 

Simon, H. Behavioral models of rational choice. In N. Polsby (Ed.), 
Politics ar.d social life . Boston: Houghton Mifflin, 1965. 

Statistical Analysis Group in Education. Results of interviews with educa - 
tional policymakers: A step in the development of an issue identifi - 
cation system for educational policies (Interim Report). Palo Alto, 
Calif.: American Institutes for Research, 1979. 

Wild, C. L., Fortna, R., & Knapp, J. Development of a system for the 
empirical determination of issues in postsecondary education . 
Princeton, N.J.: Educational Testing Service, 1978. 



20 2<i 



APPENDIX A 



Overview of the System for Ideatifyiag and Prioritizing Issues in 
Elementary and Secondary Education 

The purpose of SAGE Task 1 was to develop a system for identifying 
and analyzing current and emerging issues in elementary and secondary 
education. The Center staff could then use the results of this effort 
in planning and in developing future data collection and reporting 
efforts. Presumably, implementing such a process will increase che 
frequency with which NCES products provide critical inputs to the 
education policyuiaking process. 

From a review of previous efforts , it became clear that debates 
have raged concerning the definition and the formulation of issues. 
The authors believe that the definition of an "issue," whether theo- 
retical or operational, is not as important as is the action to be 
taken by NCES as a result of the identified issues and data needs. 
What is important is that issues provide a link between policymaker 
needs and statistical reports that can be generated by NCES, Never- 
theless, it is useful to document the approach that was followed in 
the present study, in order to validate the statements made in the 
report concerning the importance oi different issues. 

The dictionary defines an "issue" as "(1) a point of discussion, 
debate, or dispute, and (2) a matter of wide public concern" (Morris, 
1971). Extending that definition somewhat, an issue is a question for 
which the answer is important in resolving or structuring discussions, 
debates, and disputes among individuals. By defining an issue as a 
question, we are avoiding a formulation in terms of statements, either 
as "topics of concern," such as "school finance" or "desegregation," 
or as declarations of fact. Both of these latter formulations suffer 
from being too v;igue, and they force the reader to decide or identify 
the points of debate. Questions, on the other hand, directly indicate 
an information need. 
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Having defined an issue, it is iisportaat to clarify different 
types of issues* An empirical issue is one that can be answered by 
the gathering of information. A rese " rh issue is an empirical issue 
in which the answer verifies or contradicts a scientific h3^othesis or 
theory. A policy issue may or may not be empirical, but the altema* 
tive answers or resolutions to that issue have implications for policy 
in that they support certain decisions or actions. For NCES, empirical 
policy issues are of special importance, since a pirt of the Center's 
mission is to provide the information necessary for formulating and 
evaluating federal education policy alternatives* 

Returning to the purpose of this SAGE effort, it was obvious to 
the authors that, to be useful, a document-monitoring system for iden- 
tifying and analyzing issues had to do more than aid in the identifi- 
cation of documents to be read and types of issues to be counted. A 
long li5t of issues in elementary and secondary education could be 
generated with or without ou.ch readi g by just recording "obvious" 
questions. What is of interest to NCES, though, is tae identification 
of important empirical, research, and policy issues for use in future 
plann i ng. Because of limited resources, the Center must direct its 
efforts toward a selected set of issues. Therefore, the system must 
assist not only in identifying issues but also in establishing priori- 
ties among issues and in elaborating the data needs. The following 
sections briefly describe the process by which issues were identified 
from written documents and by which priorities were established among 
these issues. 

Identifying Issues 

Given the purposes of this task and the definitions that we have 
imposed, certain limitations or constraints to the issue identification 
process must be recognized: 

(1) Any identified issue must pertain to elementary or secondary 
education. 



(2) There oust be evidence of lential dis*-greement or debate 
about the answer to the question. 

(3) It should be a natter of wide public concern. 

(4) It aiust be potentially possible for information to be 
gathered or reported that will illuminate the debate sur- 
rounding the issue. 

The issue identification process focused on a review of print^^d 
sources. To detemine important written documents for this revi^'W, 
interviews were conducted with individuals active in the national 
policymaking arena. The following publications were identified as 
being key sources and were subsequently monitored by the project 
staff: 

Education Daily 
Harvard Educational Review 
Phi Delta Kappan 
Education U.S.A. 
National Journal 
Congressional Quarterly 
Public Opinion 
School Review 

In view of the need for identifying current issues, as well as the 
large volume of material to be reviewed, the document monitoring was 
limited to the period between 1 Jrie 1978 and 30 April 1979. Future 
replications of this effort might focus on a similar time period every 
two or three years to update the Center's issue-base for plaiming data 
collection and reporting activities. 

Guidelines were developed to assist the staff in identifying 
issue material and formulating issue questions embodied in the docu- 
ments. In particular, certain indicator- were used to sign2,l the 
presence of an issue question. These included (1) emotion words 
(e.g., "problem," "issue," "debate") or ^alue judgments; (2) reports 
of current or imminent changes in aspects of the education system; 
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(3) reports of proposed changes in aspects of the education system, 
and (4) the existence of causal statements, beliefs, or findings. The 
presence of any one of these indicators provided the basis for inspect- 
ing and aiulyzing the natu;:e of the concern more closely. Following 
this analysis, a foiTnat was selected for the wording of the issue 
question. The entire process is displayed in Figure A-1 and :an be 
considered to be the operational definition of an issue as used in 
this project. 

After recording the issue question on an "issue card," the reader 
attempted to assign the issue to a specific location within a taxonomy. 
The purpose of this assignmeat was threefold. First, for many poten- 
tially important issues being identified, the taxonomy provided a 
convenient mechanism for charting the relationships among them. 
Second, assignment to the taxonomy afforded some f^-rther insight into 
the nature of the debate. Third, review of the issue cards in a part 
of the taxonomy supplied the reader with an overview of the current 
debates in that particular area of concern. The taxonomy for record- 
ing and indexing issues in documents is presented in Figure A-2. 

An overview of the entire document review process appears in 
Fi^:ure A-3. It provides a listing of each of the steps followed by 
the readers. 
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Although the previously mentioned taxonomy provided a deductively 
developed categorization, it was recognized that an inductively-developed 
taxonomy would be more closely tied to current issues and concerns. 
Therefore, the staff followed the procedures listed below. Ir should 
be noted that these are a modification of guidelines set forth by 
Flanagan (1954, 1974) for analyzing critical incidents. (Indeed, the 
entire process of issue identification can be viewed as a modification 
of the critical incident technique.) 

1. Select a general frame of reference. 

2. Sort a sample of issue cards into a few piles in accordance 
with th^ selected frame of referancs. 
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Materials are 
received by 
Sack and 
entered Inco 
SAGE library 
sysc^ 



Each document 




Document 




assigned I.D. 




Is placed 




number with 




in stock 




cask . *aflx 




for review 




(e.g. cask 10 




and Issue 




might be 10-53) 




extraction 





Read for Issues. 
Star (*) to note 
If documerit Is to 
be set aside 
for review 




Write abstract. 
Abstract and APA- 
style reference 
are attached to 
document 



Subtasks In Issue extraction 



A. 

B. 
C. 

D. 



F. 



U. 

I. 
J. 
K. 



Read document locking for flags (see "guide for Identifying IssuCb") . 

Draw box around text containing statement of Issue where flag occurs. 

Identify flag type (e.g.» A troubling condition Is recognized, Imminent change 
recognized* change proposed » causal statement). 

Using guide translate Issue into standard format (e.g.» Is It true that.«.?). 
If statement too vague see *'£.3." below. Record Issue-questlop and document 
number on f^^ecard as In illustration. 



Using taxo'nomy for recording issues (a portion of the form is illustrated below) 

1. Record document number on form. 

2. For each issue statement recorded (one per index card) write index card 
number in appropriate cell of taxonomy matrix. 

3.. For issues too vaguely stated for explicit identification of cell assign 
to general category tuch as ''Population Served." Translate into: ^*Is 
there evidence that is problematic?" 

Record issue-area code into card. For this example; 

Major category is lA. Population served; subgroups 
Minor category is 5. other. 

Type of issue is w. A troubling (problematic) condition is recognized. 



Therefore issue code is IA5 w. 



Continue through document identifying issues and translating them into coirmon 
form» filling out the taxonomy form and index cards until the document is 
completed. 

Decide if this document should be reviewed prior to drafting policy pa^er. 
If so put * on upper right hand corner of cover sheet for document which will 
have complete reference and abstract of It for typist. 

Return document to shelf for typist with cover sheet attached. 

File any index cards by issue codes. 

File taxonomy forms by form number. 



Abstract 

and reference 

are typed 

ing ECRM type- 
sphere 



Typed reference 
and abstract are 
sent to SYSTEM 6 



7 V 



Document is 
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Document is 
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separately 
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Figure A-1. Overview of document review. 



iESTCOPYAVAfLMtLE 



3u 



' * Mm' t^o'* word a 

(i^'.gr, "prubKV \ • 

"Issue". **d Ucu«8tu n", 
ur "Jebaie"). or 
value J udgcB^nt 
(e.p.., hack Co basics 
la a alstaka, cunfldenttf 
Is l»eliig underlined, 
tiuublliig cundltloiis 
exist) 

* * • ■ In aspects 

uf the education 1 

sya tew reported tu 
be occur i ng or 
fluent (e.g., decllu- 
Inis test scores, de- 
creasing ssClsfacilnii 
wlCli education system) 

111. Pru|mat'd clianyes _l n 

a niiccts of the edticat- — < 
Ion syatew (e.g.^eval- 
nation of textbooks^ 



ANALYZE TO DEIEKMINb 
NATURE UF CONCEKN* 



A troubling current 
Londlllou Is rccof^iilzed.^ 



(:M<X)SE FcmMAT 



Is It true that... 

a. Che "troubling" 
cuudli Inn exists 7 
there Is Justify 
Icallun fur consid- 
ering It truuhllng? 
the change Is l«> 
■ Inont? — 



i iMmlnent chaaj^c In 
aspect uf education 
aystew Is lecognlzed. 



IV. Ca<is.il stateaents , 
f indings 
(effectiveness of 
X depends on x 
lends to Y, If x 
then y) 



Change In aspect uf 
educ.it Ion aysten Is 
proposed. 



Aspect uf education 
systeM Is described 
or explfilncd with 
Iwpl ICfit lulls f (ir 
change or InproveoMMit . 




II. Would/will benefits of 
proposed or reported 
changes outweigh ci>flts7\ 



Hi. 



I AbfiKSS I 

I impoktanceI 



is It true that. 
' a. the cxplanatlun c»r 
dus( riptltm Is 
occurote? 
^b. the Ir.pllcatinn that 
change sliuuld occur 
follfws logically 
given uvA.Vnce that 
Might b cullcctud? 



(tentative conslderst luns) 

Assuiiing It Is true, would 
^It Invalldste or validate 
the prt'ialses of existing 
educational pullcles ur 
practlies at federal, state 
or lucal levels? 

Would It alert pel cyiaakers, 
adalnlstiaturs or researchers 
to critical prublcas? 



is the benefit cost ratio likely 
'to 111' cKtit^Niely high or low? 



Is this change likely tu uccur? 



* Many paiagrapha will begin with s general topic sentence which contains a flag. Continue reading the paragrtiph 
to extract the laoat aperlfic deacr Ipt lon(u) of cunccrn(a) the authur(s) iMikes. 

If a sentence cuntalns wore than a single flag tunslder Multlt>lo k'uds uf cum ern and aulllplc Isnue questluns. 



Figure A-2. Guide for identifying issues in documents, 
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Figure A-3. Taxonomy for recording and indexing issues in documents. 
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3. Formulate tentative headings for the major categories. 

4. Sort additional issue cards into these major areas and 
setting up new subcategories as necessary. (During this 
process, all issue cards that were so similar that they 
would remain together regardless of changes in category 
definitions were clipped together and treated as one unit.) 

5. Prepare tentative definitions for major headings and gen-* 
eralized issue questions for each of the main categories of 
issues. 

6. Redefine major areas and issue questions as necessary while 
issue cards are being classified. 

7. Have an independent check made on the classification of 
issue questions. 
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